[Yanel-dev] User management search implementation
Balz Schreier
balz.schreier at gmail.com
Mon Apr 18 18:04:41 CEST 2011
SOLVED!
Root Cause: if you use "split paths" in the user repository, the current
implementation fails.
Solution Explanation:
In the constructor of the YarepUsersIterator, there is a FOR loop:
for(Node n : userNodes) {
if(n.isResource()) {
userNodeList.add(n);
}
}
if you look up the method node.isResource() in the VirtualFileSystemNode you
see that isResource() uses the node.getType() method in order to compare it
to NodeType.RESOURCE.
getType() looks like this at the moment:
public int getType() throws RepositoryException {
if (getRepository().getMap().isCollection(new Path(path))) {
return NodeType.COLLECTION;
} else if (getRepository().getMap().isResource(new Path(path))) {
return NodeType.RESOURCE;
} else {
return -1;
}
}
you see that the variable PATH is just used as is but in a "split path"
repository like in my case, this does not work at all.
So the fix is to introduce split checks also in this method.
The new patch for Yarep is attached.
You can therefore replace my last patch regarding yarep with this one!
Can we please look into this tomorrow?
Please note that VirtualFileSystemNode.getNodes() got fixed too (but is
backwards compatible).
Cheers
Balz
On Mon, Apr 18, 2011 at 5:16 PM, Balz Schreier <balz.schreier at gmail.com>wrote:
> hi,
> in the meanwhile I found out the following:
> in the ListUsersResource class there is an initVars() method.
> In there is a line:
>
> userManager.getAllUsers();
>
> this returns an iterator with no items (hasNext() is false).
>
> On the other hand,
>
> userManager.getUserCount();
>
> returns the correct number of users (and is also displayed on the User List
> Page).
>
> I'm investigating further...
> Cheers
> Balz
>
> On Mon, Apr 18, 2011 at 4:21 PM, Balz Schreier <balz.schreier at gmail.com>wrote:
>
>> hi,
>> I have now applied the configuration so that users should get searchable
>> on the User Mgmt page.
>> But no users appear at all. I even registered a new user (so that it gets
>> certainly indexed) but that didn't help either.
>>
>> In general: is a reindexing necessary of the identites repository after
>> applying this configuration?
>>
>> The Index is there, I can see it (fulltext and properties index).
>>
>> Current config is:
>>
>> <?xml version="1.0"?>
>>
>> <repository class="com.zwischengas.yarep.ZGVirtualFileSystemRepository">
>>
>> <name>identities</name>
>>
>> <splitpath depth="2" length="2" escape="+">
>>
>> <include path="/users/" /> <!-- trailing slash is required!! -->
>>
>> <include path="/aliases/" /> <!-- trailing slash is required!! -->
>>
>> </splitpath>
>>
>>
>> <meta src="../access-control-meta"/>
>>
>> <content src="../access-control">
>>
>> <ignore pattern=".*\.yarep"/>
>>
>> <ignore pattern=".*\.svn"/>
>>
>> <ignore pattern=".*\.svn.*"/>
>>
>> <ignore pattern="\.svn"/>
>>
>> <ignore pattern="\.svn.*"/>
>>
>> <ignore pattern=".*\.DS_Store"/>
>>
>> <ignore pattern=".*\.swp"/>
>>
>> </content>
>>
>>
>> <!-- Search index config -->
>>
>> <s:search-index
>>
>> xmlns:s="http://www.wyona.org/yarep/search/2.0"
>>
>> indexer-class="org.wyona.yarep.impl.search.lucene.LuceneIndexer"
>>
>> searcher-class="org.wyona.yarep.impl.search.lucene.LuceneSearcher">
>>
>> <index-location file="../../yarep-search/users-index"/>
>>
>> <repo-auto-index-fulltext boolean="true"/>
>>
>> <repo-auto-index-properties boolean="true"/>
>>
>> <lucene>
>>
>> <!-- <local-tika-config file="tika-config.xml"/> -->
>>
>> <fulltext-analyzer class=
>> "org.apache.lucene.analysis.standard.StandardAnalyzer"/>
>>
>> <property-analyzer class=
>> "org.apache.lucene.analysis.WhitespaceAnalyzer"/>
>>
>> <write-lock-timeout ms="3000"/>
>>
>> </lucene>
>>
>> </s:search-index>
>>
>> </repository>
>>
>> Cheers
>> Balz
>>
>>
>> On Wed, Jan 5, 2011 at 1:34 PM, Michael Wechner <
>> michael.wechner at wyona.com> wrote:
>>
>>> Hi Balz
>>>
>>>
>>> On 1/5/11 1:24 PM, Balz Schreier wrote:
>>>
>>> Hi Michael,
>>> do you have a wiki page showing the necessary configuration steps in a
>>> realm in order to make use of this new feature?
>>> because without any modification the user XMLs are not indexed.
>>>
>>>
>>> Have a look at
>>>
>>>
>>> src/realms/from-scratch-realm-template/config/ac-identities-repository.xml
>>>
>>> and update your custom config accordingly
>>>
>>>
>>> I would also like to understand in a bit more detail what is going on
>>> under the hood:
>>> - is it using the same index as the default index from yanel?
>>>
>>>
>>> each repository can have its own index, but see below
>>>
>>> if not, where is the index stored?
>>>
>>>
>>> this depends on your configuration, see again
>>>
>>>
>>> src/realms/from-scratch-realm-template/config/ac-identities-repository.xml
>>>
>>> - can it affect the search functionality of the default index? i just
>>> want to be sure that search results from the default index do not contain
>>> user objects.
>>>
>>>
>>> if you do it as the from scratch realm does it, where each repository has
>>> its own index, then no
>>>
>>> HTH
>>>
>>> Michael
>>>
>>>
>>> cheers
>>> balz
>>>
>>> On Tue, Dec 28, 2010 at 10:53 PM, Michael Wechner <
>>> michael.wechner at wyona.com> wrote:
>>>
>>>> Hi Cedric
>>>>
>>>> Thanks again for your patches. I have committed them with some minor
>>>> changes,
>>>> whereas I first had to fix the YarepItem class because it was not
>>>> closing the OutputStream
>>>> and hence the indexing was never triggered when updating a user.
>>>>
>>>> One can now test it at
>>>>
>>>>
>>>> http://127.0.0.1:8080/yanel/from-scratch-realm/yanel/admin/list-users.html?query=alice
>>>>
>>>> whereas you need to update/re-build Yanel first and then do some
>>>> modifications, by changing
>>>> for example the name of these two users
>>>>
>>>> http://127.0.0.1:8080/yanel/from-scratch-realm/yanel/users/lenya.html
>>>> http://127.0.0.1:8080/yanel/from-scratch-realm/yanel/users/alice.html
>>>>
>>>> The user interface needs some more work, but maybe we can collect some
>>>> feedback first
>>>> before we continue with more improvements.
>>>>
>>>> Thanks
>>>>
>>>> Michael
>>>>
>>>>
>>>> On 10/21/10 4:18 PM, Cedric Staub wrote:
>>>>
>>>>> Hello everyone
>>>>>
>>>>> In short: I built a search feature for the user management page.
>>>>>
>>>>> The current implementation of Yanel's user management resource does not
>>>>> support searching for users, which can be a bit annoying if you have a
>>>>> realm with a lot of users. Since Yarep already has an interface that
>>>>> allows to search for nodes, I built a search feature for the user
>>>>> management page on top of that.
>>>>>
>>>>> There is one limitation to this approach: The search feature is subject
>>>>> to the limitations of the underlying implementation, e.g. if your users
>>>>> repository does not support searching it won't work. I suggest the
>>>>> VirtualFileSystemRepository implementation, which can be configured to
>>>>> use a Lucene search index.
>>>>>
>>>>> Attached are a series of patches for review. If anyone has suggestions
>>>>> or ideas for improvements just reply to this email ;-). I plan to
>>>>> improve the code and do some more testing and then file a bug report
>>>>> later in order to get it commited.
>>>>>
>>>>> Have a nice day
>>>>> Cedric
>>>>>
>>>>
>>>> --
>>>> Yanel-development mailing list Yanel-development at wyona.com
>>>> http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Yanel-development mailing list Yanel-development at wyona.com
>>> http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wyona.org/pipermail/yanel-development/attachments/20110418/fa127b29/attachment-0001.html>
-------------- next part --------------
Index: src/impl/java/org/wyona/yarep/impl/repo/vfs/VirtualFileSystemNode.java
===================================================================
--- src/impl/java/org/wyona/yarep/impl/repo/vfs/VirtualFileSystemNode.java (revision 57761)
+++ src/impl/java/org/wyona/yarep/impl/repo/vfs/VirtualFileSystemNode.java (working copy)
@@ -11,23 +11,17 @@
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
+import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Calendar;
import java.util.Date;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedHashMap;
+import java.util.List;
+import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
-import org.apache.commons.io.FileUtils;
import org.apache.log4j.Logger;
-
-import org.apache.lucene.document.Document;
-import org.apache.lucene.document.Field;
-import org.apache.lucene.document.Field.Index;
-import org.apache.lucene.document.Field.Store;
-import org.apache.lucene.index.IndexWriter;
-
import org.wyona.yarep.core.NoSuchRevisionException;
import org.wyona.yarep.core.Node;
import org.wyona.yarep.core.NodeStateException;
@@ -99,10 +93,10 @@
protected void init() throws RepositoryException {
this.contentDir = getRepository().getContentDir();
- this.contentFile = new File(this.contentDir, getRepository().splitPath(this.uuid));
+ this.contentFile = new File(this.contentDir, SplitPathConfig.splitPathIfRequired(this.uuid, getRepository().getSplitPathConfig()));
this.backupContentFile = new File(this.contentDir, this.uuid);
- String metauuid = getRepository().splitPath(uuid + META_DIR_SUFFIX);
+ String metauuid = SplitPathConfig.splitPathIfRequired(uuid + META_DIR_SUFFIX, getRepository().getSplitPathConfig());
if (getRepository().getMetaDir() != null) {
this.metaDir = new File(getRepository().getMetaDir(), metauuid);
this.backupMetaDir = new File(getRepository().getMetaDir(), uuid + META_DIR_SUFFIX);
@@ -293,28 +287,174 @@
}
}
- /**
- * @see org.wyona.yarep.core.Node#getNodes()
- */
public Node[] getNodes() throws RepositoryException {
- //log.debug("Get nodes from repository: " + getRepository().getClass().getName() + ", " + getRepository().getConfigFile());
- Path[] childPaths;
+ log.debug("+++ Node.getNodes() was called! Node Path = "+getPath());
+ return getNodes(null);
+ }
+
+ public Node[] getNodes(FileFilter fileFilter) throws RepositoryException {
+ Node[] childNodes = new Node[0];
+
+ List<String> listOfPaths = getNodesAsPathStrings(fileFilter);
+
+ if (!listOfPaths.isEmpty()) {
+ log.warn("Node[] getNodes(): Converting all Node paths into Node objects and putting them into an array");
+ childNodes = new Node[listOfPaths.size()];
+ int i = 0;
+ for (String path: listOfPaths) {
+ childNodes[i] = this.repository.getNode(path);
+ i++;
+ }
+ log.warn("Getting all "+childNodes.length+" children nodes: DONE");
+
+ }
+
+ return childNodes;
+ }
+
+ public List<String> getNodesAsPathStrings(FileFilter fileFilter) throws RepositoryException {
+ List<String> childPaths = new ArrayList<String>();
+ Path[] nonSplitChildren = null;
if (getRepository().isSplitPathEnabled()) {
- // TODO: Unsplit paths
- log.warn("TODO: Unsplit paths...");
- childPaths = getRepository().getMap().getChildren(new Path(this.path));
+
+ File startingDirectory = new File(contentDir + path);
+ log.debug("startingDirectory= "+startingDirectory);
+ if (!startingDirectory.exists()) {
+ log.debug("No such file or directory: " + startingDirectory);
+ //childPaths = new Path[0];
+
+ } else if (startingDirectory.isFile()) {
+ log.debug("Can not get children form a file! : " + startingDirectory);
+ //childPaths = new Path[0];
+ } else {
+ // only if the current path belongs to a "include" path for the split paths, we search ALL children, otherwise just the direct children
+ log.debug("Checking whether this path is include: "+path);
+ if (!SplitPathConfig.isIncludePath(path, getRepository().getSplitPathConfig())) {
+ log.warn("Path is not split: "+path);
+ nonSplitChildren = getRepository().getMap().getChildren(new Path(path));
+
+ } else {
+ log.warn("+++ Path is split! Path = "+path+" --> Calling getAllFiles()...");
+ long start = System.currentTimeMillis();
+ List<File> allChildren = getAllFiles(startingDirectory, fileFilter); // get all children from the location "this.path"
+ log.warn("+++ getAllFiles() took "+(System.currentTimeMillis()-start)+"ms");
+ log.warn("+++ Number of children in split path: " + allChildren.size() + " (" + startingDirectory + ")");
+ List<String> validChildrenPaths = new ArrayList<String>();
+ String fileSepForRegEx = File.separator;
+ if (fileSepForRegEx.equals("\\")) {
+ fileSepForRegEx = "\\\\"; // this is a double backslash, used for the regex later
+ }
+
+ for (File child: allChildren) {
+ String childFullPath = child.getAbsolutePath().replaceAll(fileSepForRegEx, "/"); // whatever the file separator was, yarep uses "/"
+ log.debug("startingDirectory= "+startingDirectory);
+ log.debug("child = "+child.getAbsolutePath());
+ if (childFullPath.startsWith(contentDir.getAbsolutePath())) {
+ String pathAndNode = childFullPath.substring(contentDir.getAbsolutePath().length()); // = starts with this.path, e.g. "/path/node"
+ log.debug("1. pathAndNode = "+pathAndNode);
+ String unsplitPathAndNode = SplitPathConfig.unsplitPathIfRequired(pathAndNode, getRepository().getSplitPathConfig());
+ String unsplitNode = unsplitPathAndNode.substring(path.length()); // we remove the path (prefix) from e.g. "/path/node.xml" -> "/node.xml"
+ log.debug("2. unsplitNode = "+unsplitNode);
+ boolean ignore = ((org.wyona.yarep.impl.VFileSystemMapImpl) getRepository().getMap()).ignorePath(unsplitNode);
+ if (!ignore) {
+ String newPath = null;
+ if (path.equals(File.separator)) {
+ newPath = unsplitNode;
+ if (!newPath.startsWith("/")) {
+ newPath = "/" + newPath;
+ }
+ log.debug("path was File separator. newPath = "+newPath);
+
+ } else if (path.toString().endsWith(File.separator) || path.toString().endsWith("/")) {
+ newPath = path.substring(0,path.length()-1);
+ if (!unsplitNode.startsWith("/")) {
+ unsplitNode = "/" + unsplitNode;
+ }
+ newPath = path + unsplitNode;
+ log.debug("path ended with a /. newPath = "+newPath.toString());
+ } else {
+ // NOTE: Do not use File.separator here, because it's the repository path and not the Operating System's File System path
+ if (!unsplitNode.startsWith("/")) {
+ unsplitNode = "/" + unsplitNode;
+ }
+ newPath = path + unsplitNode;
+ log.debug("path does not end with a /. newPath = "+newPath.toString());
+ }
+ validChildrenPaths.add(newPath);
+ log.debug("getNodes('"+path+"'): ADDED: "+unsplitNode);
+ } else {
+ log.warn("getNodes('"+path+"'): IGNORED: "+unsplitNode);
+ }
+ } else {
+ log.error("Something is wrong: children are not within parents!");
+ }
+ }
+ log.warn("=== All children found: "+validChildrenPaths.size()+" ===");
+ //childPaths = validChildrenPaths.toArray(new Path[validChildrenPaths.size()]);
+ childPaths = validChildrenPaths;
+ } // end if-else: include or not
+
+ }
+
+
} else {
- childPaths = getRepository().getMap().getChildren(new Path(this.path));
+ // split path is not enabled
+ nonSplitChildren = getRepository().getMap().getChildren(new Path(this.path));
}
- Node[] childNodes = new Node[childPaths.length];
- for (int i=0; i<childPaths.length; i++) {
- childNodes[i] = this.repository.getNode(childPaths[i].toString());
+ if (childPaths.isEmpty()) {
+ if (nonSplitChildren != null) {
+ for (Path path: nonSplitChildren) {
+ childPaths.add(path.toString());
+ }
+ }
}
- return childNodes;
+ return childPaths;
}
-
+
/**
+ *
+ * @param dir directory to get all files from
+ * @param fileFilter you can pass in NULL if you don't want to filter for certain file patterns
+ * @return List of Files (directories are NOT in the list!)
+ */
+ private List<File> getAllFiles(File dir, FileFilter fileFilter) {
+ List<File> result = new ArrayList<File>();
+ if (dir.isDirectory()) {
+ long start = System.currentTimeMillis();
+ File[] filesAndDirsArray = null;
+ if (fileFilter != null) {
+ filesAndDirsArray = dir.listFiles(fileFilter);
+ } else {
+ filesAndDirsArray = dir.listFiles();
+ }
+ // A warning shall be logged if something takes longer than 10 seconds!
+ long delta = System.currentTimeMillis()-start;
+ if (delta > 10000) {
+ log.warn("List created in "+(System.currentTimeMillis()-start)+"ms"+" : "+dir.getAbsolutePath());
+ }
+
+ start = System.currentTimeMillis();
+ List<File> filesAndDirs = Arrays.asList(filesAndDirsArray);
+ start = System.currentTimeMillis();
+ for (File file : filesAndDirs) {
+ if (file.isFile()) {
+ result.add(file); // because of the xmlFilter above, only xml files in the list
+ } else {
+ List<File> deeperList = getAllFiles(file, fileFilter);
+ result.addAll(deeperList);
+ }
+ }
+ // A warning shall be logged if something takes longer than 10 seconds!
+ delta = System.currentTimeMillis()-start;
+ if (delta > 10000) {
+ log.warn("Looping took "+(System.currentTimeMillis()-start)+"ms"+" : "+dir.getAbsolutePath());
+ }
+ }
+ return result;
+ }
+
+ /**
* @see org.wyona.yarep.core.Node#addNode(java.lang.String, int)
*/
public Node addNode(String name, int type) throws RepositoryException {
@@ -326,7 +466,7 @@
if (this.repository.existsNode(newPath)) {
throw new RepositoryException("Node exists already: " + newPath);
}
- UID uid = getRepository().getMap().create(new Path(getRepository().splitPath(newPath)), type);
+ UID uid = getRepository().getMap().create(new Path(SplitPathConfig.splitPathIfRequired(newPath, getRepository().getSplitPathConfig())), type);
// create file:
File file = new File(this.contentDir, uid.toString());
try {
@@ -676,13 +816,17 @@
*
*/
public int getType() throws RepositoryException {
+ String maybeSplitPath = SplitPathConfig.splitPathIfRequired(path, getRepository().getSplitPathConfig());
if (getRepository().getMap().isCollection(new Path(path))) {
return NodeType.COLLECTION;
} else if (getRepository().getMap().isResource(new Path(path))) {
return NodeType.RESOURCE;
+ } else if (getRepository().getMap().isResource(new Path(maybeSplitPath))) {
+ return NodeType.RESOURCE;
} else {
return -1;
}
+
}
/**
Index: src/impl/java/org/wyona/yarep/impl/repo/vfs/VirtualFileSystemRepository.java
===================================================================
--- src/impl/java/org/wyona/yarep/impl/repo/vfs/VirtualFileSystemRepository.java (revision 57761)
+++ src/impl/java/org/wyona/yarep/impl/repo/vfs/VirtualFileSystemRepository.java (working copy)
@@ -115,11 +115,7 @@
// Configuration parameters of the <splitpath ...> element
private boolean splitPathEnabled = false;
- private int splitparts = 0;
- private int splitlength = 0;
- private String DEFAULT_DUMMY_SEPARATOR_VALUE = "-";
- private String dummySeparator = DEFAULT_DUMMY_SEPARATOR_VALUE;
- private String[] includepaths = {};
+ private SplitPathConfig splitPathConfig = new SplitPathConfig();
/**
*
@@ -213,23 +209,22 @@
Configuration splitConfig = config.getChild("splitpath", false);
if (splitConfig != null) {
splitPathEnabled = true;
- String depth = splitConfig.getAttribute("depth", "0");
- splitparts = Integer.parseInt(depth);
+ splitPathConfig.setSplitparts(Integer.parseInt(splitConfig.getAttribute("depth", "0")));
+ splitPathConfig.setSplitlength(Integer.parseInt(splitConfig.getAttribute("length", "0")));
- String length;
- length = splitConfig.getAttribute("length", "0");
- splitlength = Integer.parseInt(length);
-
- dummySeparator = splitConfig.getAttribute("escape", DEFAULT_DUMMY_SEPARATOR_VALUE);
-
- int c = splitConfig.getChildren("include").length;
- int i = 0;
- if (c > 0) {
- includepaths = new String[c];
+ int numberOfIncludePaths = splitConfig.getChildren("include").length;
+ if (numberOfIncludePaths > 0) {
+ String[] includepaths = new String[numberOfIncludePaths];
+ int i = 0;
for (Configuration include : splitConfig.getChildren("include")) {
includepaths[i++] = include.getAttribute("path");
}
+ splitPathConfig.setIncludepaths(includepaths);
}
+ // NOTE: This repository uses the VFileSystemMapImpl: But we do not tell the map about the splitting.
+ // Splitting is known to this repository and the node only.
+ // ((org.wyona.yarep.impl.VFileSystemMapImpl) map).setSplitPathConfig(splitPathConfig);
+ // ((org.wyona.yarep.impl.VFileSystemMapImpl) map).setSplitPathEnabled(true);
}
} catch (Exception e) {
log.error(e.toString());
@@ -296,7 +291,6 @@
* @see org.wyona.yarep.core.Repository#exists(org.wyona.yarep.core.Path)
*/
public boolean exists(Path path) throws RepositoryException {
- log.warn("DEPRECATED");
return existsNode(path.toString());
}
@@ -401,7 +395,8 @@
path = path.substring(0, path.length() - 1);
}
if (splitPathEnabled) {
- return map.exists(new Path(splitPath(path))) || map.exists(new Path(path)); // INFO: The OR is because of backwards compatibility in case that a node exists with an unsplitted path, because it has not been migrated yet (which can happen if it has only been read so far, but never written since introducing the split path configuration)
+ String maybeSplit = SplitPathConfig.splitPathIfRequired(path, splitPathConfig);
+ return map.exists(new Path(maybeSplit)) || map.exists(new Path(path));
} else {
return map.exists(new Path(path));
}
@@ -416,7 +411,8 @@
path = path.substring(0, path.length() - 1);
}
- if ((splitPathEnabled && map.exists(new Path(splitPath(path)))) || map.exists(new Path(path))) {
+ String maybeSplit = SplitPathConfig.splitPathIfRequired(path, splitPathConfig);
+ if ((splitPathEnabled && map.exists(new Path(maybeSplit))) || map.exists(new Path(path))) {
String uuid = new UID(path).toString();
return new VirtualFileSystemNode(this, path, uuid);
} else {
@@ -609,98 +605,20 @@
return true;
}
- /**
- * Splits a String such that the result can be used as a repo path for a tree-like repo structure.
- *
- * This method splits off n strings (where n = parts) of length partlength, e.g. if
- * splitPath("ec2c0c02-1d7d-4a21-8a39-68f9f72dea09", 3, 4) is called, then:
- * in: ec2c0c02-1d7d-4a21-8a39-68f9f72dea09, 3, 4
- * out: ec2c/0c02/-1d7/d-4a21-8a39-68f9f72dea09
- *
- * If the strings length is shorter than parts * partslength, then as many
- * parts as possible are split, e.g.
- * in: foobar, 2, 5
- * out: fooba/r
- * in: lorem, 3, 10
- * out: lorem
- *
- * An example with "/" characters:
- * in: /foobar/lorem/ipsum.txt, parts = 3, lenght = 3
- * out: /foo/bar/-lo/rem/ipsum.txt
- *
- * @param uuid
- * @return split uuid
- */
- String splitPath(String path) {
- // NOTE: uuid should be a full yarep path, so we can safely remove
- // the leading slash
-
- // check if the given path matches any of the include values
- // in the configuration
- boolean include = false;
- String base = "";
- for (String s : includepaths) {
- if (path.startsWith(s)) {
- include = true;
- base = s;
- break;
- }
- }
+ public SplitPathConfig getSplitPathConfig() {
+ return splitPathConfig;
+ }
- // return the path unchanged if it doesn't match
- // any of the include values
- if (!include) {
- return path;
- }
-
- // remove the leading base string, will be added again later
- path = path.substring(base.length(), path.length());
-
- // replace "/" characters where needed
- if (path.length() <= splitparts * splitlength) {
- path = path.replaceAll("/", dummySeparator);
- } else {
- path = String.format("%s%s",
- path.substring(0, splitparts * splitlength).replaceAll("/", dummySeparator),
- path.substring(splitparts * splitlength));
- }
-
- // now do the actual splitting
- int len = path.length();
- int pos = 0;
- String out = "";
-
- int partc = 0;
- int w;
- while (len > 0 && partc < splitparts) {
- partc++;
- if (len < splitlength) {
- w = len;
- } else {
- w = splitlength;
- }
- out += path.substring(pos, pos + w);
- pos += w;
- len -= w;
-
- if (len > 0) {
- out += "/";
- }
- }
-
- // append remainder
- if (len > 0) {
- out += path.substring(pos, pos + len);
- }
-
- // finally, add the leading zero again and return the new path
- return base + out;
+ public void setSplitPathConfig(SplitPathConfig splitPathConfig) {
+ this.splitPathConfig = splitPathConfig;
}
- /**
- * Check whether split path is enabled and make this available to classes within this package
- */
- boolean isSplitPathEnabled() {
+ public boolean isSplitPathEnabled() {
return splitPathEnabled;
}
+
+ public void setSplitPathEnabled(boolean splitPathEnabled) {
+ this.splitPathEnabled = splitPathEnabled;
+ }
+
}
Index: src/impl/java/org/wyona/yarep/impl/repo/vfs/SplitPathConfig.java
===================================================================
--- src/impl/java/org/wyona/yarep/impl/repo/vfs/SplitPathConfig.java (revision 0)
+++ src/impl/java/org/wyona/yarep/impl/repo/vfs/SplitPathConfig.java (revision 0)
@@ -0,0 +1,266 @@
+package org.wyona.yarep.impl.repo.vfs;
+
+public class SplitPathConfig {
+ private String[] includepaths = new String[0];
+ private int splitparts;
+ private int splitlength;
+ private String escapeChar = "+";
+
+ /**
+ * Split Path Configuration:
+ * @param includepaths : if an includepath is configured, e.g. "/base/" then everything after "/base"/ gets splitted, except for the file extension!
+ * @param splitparts max subdirectories from the baserpath: e.g. base/xx/xx/xx/xx/restofpath : splitparts = 4
+ * @param splitlength the length of the additional subdirectories, e.g. base/xx/xx/xx/xx/restofpath : length=2
+ */
+ public SplitPathConfig(String[] includepaths, int splitparts, int splitlength) {
+ this.includepaths = includepaths;
+ this.splitparts = splitparts;
+ this.splitlength = splitlength;
+ }
+
+ /**
+ * Creates a config without any include paths
+ */
+ public SplitPathConfig() {
+ }
+
+ /**
+ * Splits a String such that the result can be used as a repo path for a tree-like repo structure.
+ * The original string can be shorter than splitparts * partlength
+ * The file extension does not get split.
+ * Slashes in the original string are escaped if they are in the range of the split area (splitparts * partlength):
+ * The esacpe character is currently hardcoded and is "+": a slash gets replaced by "+-" and the plus itself by "++".
+ *
+ * Example for splitparts = 2 and partlength = 3:
+ * hugoboss.xml --> hug/obo/ss.xml
+ * hugo.xml --> hug/o.xml
+ * hug.xml --> hug.xml
+ * hugo/boss.xml --> gets stored as hug/o+-/boss.xml
+ * hug/oboss.xml --> gets stored as hug/+-o/boss.xml
+ * hugo+boss.xml --> gets stored as hug/o++/boss.xml
+ * hugobos/s.xml --> gets stored as hug/obo/s/s.xml
+ * hugobo/ss.xml --> (special case: actually hugobo could be split into hug/obo/ ... but as the next character in the original string is a "/", we do not allow this in the split string, because "//" can not get unsplit anymore.
+ * So this becomes hug/obo/+-ss.xml
+ * And a mixed example with the escape character:
+ * hu/go+bo/ss.xml --> hu+/-go/++bo/ss.xml (Note that the first slash in the original string got replaced by +- and then the actual slash has been inserted!)
+ *
+ * If the strings length is shorter than parts * partslength, then as many
+ * parts as possible are split, e.g.
+ * in: foobar, 2, 5
+ * out: fooba/r
+ * in: lorem, 3, 10
+ * out: lorem
+ *
+ * An example with "/" characters:
+ * in: /foobar/lorem/ipsum.txt, parts = 3, lenght = 3
+ * out: /foo/bar/-lo/rem/ipsum.txt
+ *
+ * @param path usually the full yarep path of a node, e.g. "/users/chucknorris.xml" or if you have the data repository at "/data" and you add a node to this repository, the path is relative to this repository path, e.g. "/pictures/..." and not "/data/pictures/...
+ * @return split path according to the configured rules
+ */
+ public static String splitPathIfRequired(String path, SplitPathConfig splitPathConfig) {
+ // NOTE: uuid should be a full yarep path, so we can safely remove
+ // the leading slash
+
+ // check if the given path matches any of the include values
+ // in the configuration
+ boolean include = false;
+ String base = null;
+ for (String s : splitPathConfig.getIncludepaths()) {
+ if (path.startsWith(s)) {
+ include = true;
+ base = s;
+ break;
+ }
+ }
+
+ // return the path unchanged if it doesn't match
+ // any of the include values
+ if (!include) {
+ return path;
+ }
+
+ // remove the leading base string, will be added again later
+ path = path.substring(base.length(), path.length());
+ // we do not want to split the file ending (e.g. ".xml")
+ String suffix = "";
+ if (path.contains(".")) {
+ suffix = path.substring(path.lastIndexOf("."));
+ path = path.substring(0, path.lastIndexOf("."));
+ }
+ int splitparts = splitPathConfig.getSplitparts();
+ int splitlength = splitPathConfig.getSplitlength();
+
+ // replace "/" characters where needed
+ if (path.length() <= splitparts * splitlength) {
+ path = path.replaceAll("\\+", "++");
+ path = path.replaceAll("/", "+-");
+ } else {
+ path = path.replaceAll("\\+", "++");
+ path = String.format("%s%s",
+ path.substring(0, splitparts * splitlength).replaceAll("/", "+-"),
+ path.substring(splitparts * splitlength));
+ }
+
+ // now do the actual splitting
+ StringBuffer splitPath = new StringBuffer(path);
+ int slashIndex = splitlength;
+ int numberOfSlashesInserted = 0;
+ // slashindex < path length + number of already inserted slahes. by each inserted slash, the path gets one char bigger...
+ while (slashIndex < (path.length() + numberOfSlashesInserted) && numberOfSlashesInserted < splitparts) {
+ splitPath.insert(slashIndex, "/");
+ slashIndex = slashIndex + splitlength + 1; // +1 because the inserted slash
+ numberOfSlashesInserted++;
+ }
+ path = base + splitPath.toString();
+
+// ORIGINAL CODE FROM PREVIOUS SPLIT IMPL : I found this too complex and hard to understand.
+// int len = path.length();
+// int pos = 0;
+// String out = "";
+//
+// int partc = 0;
+// int w;
+// while (len > 0 && partc < splitparts) {
+// partc++;
+// if (len < splitlength) {
+// w = len;
+// } else {
+// w = splitlength;
+// }
+// out += path.substring(pos, pos + w);
+// pos += w;
+// len -= w;
+//
+// if (len > 0) {
+// out += "/";
+// }
+// }
+//
+// // append remainder
+// if (len > 0) {
+// out += path.substring(pos, pos + len);
+// }
+//
+// // finally, add the leading zero again and return the new path
+// path = base + out;
+
+
+ if (path.contains("//")) {
+ path = path.replaceAll("//", "/+-");
+ }
+ // and we add the suffix again
+ path = path + suffix;
+
+ return path;
+ }
+
+ public static String unsplitPathIfRequired(String path, SplitPathConfig splitPathConfig) {
+ boolean include = false;
+ String base = "";
+ for (String s : splitPathConfig.getIncludepaths()) {
+ if (path.startsWith(s)) {
+ include = true;
+ base = s;
+ break;
+ }
+ }
+
+ if (!include) {
+ return path;
+ }
+ // remove the leading base string, will be added again later
+ path = path.substring(base.length(), path.length());
+ int splitparts = splitPathConfig.getSplitparts();
+ int splitlength = splitPathConfig.getSplitlength();
+
+ // we know that each "/" must be removed and every "+" becomes a slash.
+
+ // the area where we apply the logic is the original length (splitparts * splitlength) plus one char for each splitpart ("/")
+ int splitLength = (splitparts * splitlength)+splitparts+1;
+ if (path.length()<splitLength) {
+ splitLength = path.length();
+ }
+ // remove all slashes
+ path = path.substring(0, splitLength).replaceAll("/", "")+path.substring(splitLength);
+
+ // a simple replacement of (++ -> +) and (+- -> /) does not work because they must be replaced from left to right.
+ StringBuffer convertedPath = new StringBuffer("");
+ char current;
+ char next;
+ int i = 0;
+ for (; i < path.length()-1; i++) {
+ current = path.charAt(i);
+ next = path.charAt(i+1);
+ if (current == '+' && next == '+') {
+ convertedPath.append("+");
+ i++; // skip next char because we found a token!
+ } else if (current == '+' && next == '-') {
+ convertedPath.append("/");
+ i++; // skip next char because we found a token!
+ } else {
+ convertedPath.append(current);
+ }
+ }
+ if (i == path.length()-1) {
+ convertedPath.append(path.charAt(i));
+ }
+
+// // all original pluses are created
+// path = path.replaceAll("(\\+\\+)", "+");
+// // all original slashes are created (single + means slash)
+// path = path.replaceAll("(\\+-)", "/");
+ path = base + convertedPath.toString();
+ return path;
+ }
+
+ public static boolean isIncludePath(String path, SplitPathConfig splitPathConfig) {
+ boolean isIncludePath = false;
+ // currently the configuration of split include paths is like "/path/", therefore we have to add a slash if it is missing. otherwise it doesnot match
+ if (!path.endsWith("/")) {
+ path = path + "/";
+ }
+ for (String s : splitPathConfig.getIncludepaths()) {
+ if (path.startsWith(s)) {
+ isIncludePath = true;
+ break;
+ }
+ }
+ return isIncludePath;
+ }
+
+
+
+ public String[] getIncludepaths() {
+ return includepaths;
+ }
+
+ public void setIncludepaths(String[] includepaths) {
+ this.includepaths = includepaths;
+ }
+
+ public int getSplitparts() {
+ return splitparts;
+ }
+
+ public void setSplitparts(int splitparts) {
+ this.splitparts = splitparts;
+ }
+
+ public int getSplitlength() {
+ return splitlength;
+ }
+
+ public void setSplitlength(int splitlength) {
+ this.splitlength = splitlength;
+ }
+
+ public String getEscapeChar() {
+ return escapeChar;
+ }
+
+ public void setEscapeChar(String separator) {
+ this.escapeChar = separator;
+ }
+
+}
Index: src/impl/java/org/wyona/yarep/impl/VFileSystemMapImpl.java
===================================================================
--- src/impl/java/org/wyona/yarep/impl/VFileSystemMapImpl.java (revision 57761)
+++ src/impl/java/org/wyona/yarep/impl/VFileSystemMapImpl.java (working copy)
@@ -1,23 +1,22 @@
package org.wyona.yarep.impl;
+import java.io.File;
+import java.io.FilenameFilter;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
import org.apache.avalon.framework.configuration.Configuration;
-
+import org.apache.log4j.Category;
import org.wyona.commons.io.FileUtil;
import org.wyona.yarep.core.Map;
import org.wyona.yarep.core.Path;
import org.wyona.yarep.core.RepositoryException;
import org.wyona.yarep.core.UID;
+import org.wyona.yarep.impl.repo.vfs.SplitPathConfig;
-import java.io.File;
-import java.io.FileReader;
-import java.io.FileWriter;
-import java.io.BufferedReader;
-import java.io.FilenameFilter;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-import org.apache.log4j.Category;
-
/**
*
*/
@@ -29,7 +28,11 @@
protected Pattern[] ignorePatterns;
protected ChildrenFilter childrenFilter = new ChildrenFilter();
+ // Configuration parameters of the <splitpath ...> element
+ private boolean splitPathEnabled = false;
+ private SplitPathConfig splitPathConfig = new SplitPathConfig();
+
/**
*
*/
@@ -41,6 +44,27 @@
Configuration[] ignoreElements = mapConfig.getChildren("ignore");
setIgnorePatterns(ignoreElements);
}
+ // Read the <splitpath> configuration
+ log.debug("Reading Split Path Configuation");
+ Configuration splitConfig = mapConfig.getChild("splitpath", false);
+ if (splitConfig != null) {
+ splitPathConfig.setSplitparts(Integer.parseInt(splitConfig.getAttribute("depth", "0")));
+ splitPathConfig.setSplitlength(Integer.parseInt(splitConfig.getAttribute("length", "0")));
+ splitPathConfig.setEscapeChar(splitConfig.getAttribute("escape", "-"));
+
+ int numberOfIncludePaths = splitConfig.getChildren("include").length;
+ int i = 0;
+ if (numberOfIncludePaths > 0) {
+ String[] includepaths = new String[numberOfIncludePaths];
+ for (Configuration include : splitConfig.getChildren("include")) {
+ includepaths[i++] = include.getAttribute("path");
+ }
+ splitPathConfig.setIncludepaths(includepaths);
+ }
+ log.debug("Split Path Configuration DONE");
+ splitPathEnabled = true;
+ }
+
} catch(Exception e) {
log.error(e);
throw new RepositoryException("Could not read map configuration: "
@@ -66,15 +90,14 @@
/**
* Test if path should be ignored
+ * @return true returns true if the path can be ignored
*/
- protected boolean ignorePath(String path) {
+ public boolean ignorePath(String path) {
if (ignorePatterns != null) {
for (int i=0; i<this.ignorePatterns.length; i++) {
Matcher matcher = this.ignorePatterns[i].matcher(path);
if (matcher.matches()) {
- if (log.isDebugEnabled()) {
- log.debug(path + " matched ignore pattern " + ignorePatterns[i].pattern());
- }
+ log.debug(path + " matched ignore pattern " + ignorePatterns[i].pattern());
return true;
}
}
@@ -89,7 +112,14 @@
*
*/
public boolean isResource(Path path) throws RepositoryException {
- File file = new File(pathsDir + path.toString());
+ File file = null;
+ if (splitPathEnabled) {
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ file = new File(pathsDir + maybeSplitedPath);
+ }
+ if (file == null || !file.exists()) {
+ file = new File(pathsDir + path.toString());
+ }
return file.isFile();
}
@@ -97,14 +127,21 @@
*
*/
public boolean exists(Path path) throws RepositoryException {
- File file = new File(pathsDir + path.toString());
- // TODO: Get name of repository for debugging ...
- //log.debug("File: " + file);
- return file.exists() && !ignorePath(file.getPath());
+ File file = null;
+ if (splitPathEnabled) {
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ file = new File(pathsDir + maybeSplitedPath);
+ }
+ if (file == null || !file.exists()) {
+ file = new File(pathsDir + path.toString());
+ }
+ boolean result = file.exists() && !ignorePath(file.getPath());
+ log.debug("file.exists()="+result+": File: "+file.getPath());
+ return result;
}
/**
- *
+ * Calling this method has no effect anymore because delete is done in the storage impl!
*/
public boolean delete(Path path) throws RepositoryException {
// don't do anything because if we delete the file here, the delete
@@ -118,41 +155,124 @@
*
*/
public boolean isCollection(Path path) throws RepositoryException {
- File file = new File(pathsDir + path.toString());
+ File file = null;
+ if (splitPathEnabled) {
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ file = new File(pathsDir + maybeSplitedPath);
+ }
+ if (file == null || !file.exists()) {
+ file = new File(pathsDir + path.toString());
+ }
return file.isDirectory();
}
/**
- * Get children
+ * Get children, the path of the children includes the path of the parent!
*/
public Path[] getChildren(Path path) throws RepositoryException {
- File file = new File(pathsDir + path.toString());
- if (!file.exists()) {
- log.warn("No such file or directory: " + file);
- return new Path[0];
- }
+ // Note: path is always NOT splited, because the caller of this method does not know anything about it
+ log.debug("path = "+path.toString());
+ if (splitPathEnabled) {
+ File startingDirectory = new File(pathsDir + path.toString());
+ if (!startingDirectory.exists()) {
+ log.warn("No such file or directory: " + startingDirectory);
+ return new Path[0];
+ }
+ if (startingDirectory.isFile()) {
+ log.warn("Can not get children form a file! : " + startingDirectory);
+ return new Path[0];
+ }
+
+ List<File> allChildren = getAllFiles(startingDirectory);
+ log.debug("Number of children: " + allChildren.size() + " (" + startingDirectory + ")");
+ List<Path> validChildrenPaths = new ArrayList<Path>();
+ String fileSepForRegEx = File.separator;
+ if (fileSepForRegEx.equals("\\")) {
+ fileSepForRegEx = "\\\\"; // this is a double backslash, used for the regex later
+ }
+
+ for (File child: allChildren) {
+ String unsplitPath = child.getAbsolutePath().replaceAll(fileSepForRegEx, "/"); // whatever the file separator was, yarep uses "/"
+ log.debug("startingDirectory= "+startingDirectory);
+ log.debug("child = "+child.getAbsolutePath());
+ if (unsplitPath.startsWith(startingDirectory.getAbsolutePath())) {
+ unsplitPath = unsplitPath.substring(startingDirectory.getAbsolutePath().length());
+ log.debug("1. path to unsplit = "+unsplitPath);
+ unsplitPath = SplitPathConfig.unsplitPathIfRequired(unsplitPath, splitPathConfig);
+ log.debug("2. unsplitPath = "+unsplitPath);
+ Path newPath = null;
+ if (!ignorePath(unsplitPath)) {
+ if (path.toString().endsWith(File.separator)) {
+ newPath = new Path(path + unsplitPath);
+ log.debug("3a. Added "+newPath.toString());
+ } else {
+ // NOTE: Do not use File.separator here, because it's the repository path and not the Operating System's File System path
+ newPath = new Path(path + "/" + unsplitPath);
+ log.debug("3b. Added "+newPath.toString());
+ }
+ validChildrenPaths.add(newPath);
+ } else {
+ log.debug("ignored: "+child.getAbsolutePath());
+ }
+ } else {
+ log.error("Something is wrong: children are not within parents!");
+ }
+ }
+ Path[] childrenArray = validChildrenPaths.toArray(new Path[validChildrenPaths.size()]);
+ return childrenArray;
+
+ } else {
+ File file = new File(pathsDir + path.toString());
+ if (!file.exists()) {
+ log.warn("No such file or directory: " + file);
+ return new Path[0];
+ }
- String[] filenames = file.list(this.childrenFilter);
+ String[] filenames = file.list(this.childrenFilter);
- // NOTE: This situation should only occur if one is trying to get children for a file than a directory! One might want to consider to test first with isResource() or isCollection().
- if (filenames == null) {
- log.warn("No children: " + path + " (" + file + ")");
- return new Path[0];
- }
+ // NOTE: This situation should only occur if one is trying to get children for a file than a directory! One might want to consider to test first with isResource() or isCollection().
+ if (filenames == null) {
+ log.warn("No children: " + path + " (" + file + ")");
+ return new Path[0];
+ }
- log.debug("Number of children: " + filenames.length + " (" + file + ")");
- Path[] children = new Path[filenames.length];
- for (int i = 0;i < children.length; i++) {
- if (path.toString().endsWith(File.separator)) {
- children[i] = new Path(path + filenames[i]);
- } else {
- // NOTE: Do not use File.separator here, because it's the repository path and not the Operating System File System path
- children[i] = new Path(path + "/" + filenames[i]);
+ log.debug("Number of children: " + filenames.length + " (" + file + ")");
+ Path[] children = new Path[filenames.length];
+ for (int i = 0;i < children.length; i++) {
+ if (path.toString().endsWith(File.separator)) {
+ children[i] = new Path(path + filenames[i]);
+ } else {
+ // NOTE: Do not use File.separator here, because it's the repository path and not the Operating System File System path
+ children[i] = new Path(path + "/" + filenames[i]);
+ }
+ log.debug("Child: " + children[i]);
}
- log.debug("Child: " + children[i]);
+ return children;
+
}
- return children;
}
+
+ /**
+ *
+ * @param dir
+ * @return List of Files (directories are NOT in the list!)
+ */
+ private List<File> getAllFiles(File dir) {
+ List<File> result = new ArrayList<File>();
+ if (dir.isDirectory()) {
+ File[] filesAndDirsArray = dir.listFiles();
+ List<File> filesAndDirs = Arrays.asList(filesAndDirsArray);
+ for (File file : filesAndDirs) {
+ if (file.isFile()) {
+ result.add(file);
+ } else {
+ List<File> deeperList = getAllFiles(file);
+ result.addAll(deeperList);
+ }
+ }
+ }
+ return result;
+ }
/**
* Get UID
@@ -163,11 +283,14 @@
}
/**
- * Create UID
+ * Create UID:
*/
public synchronized UID create(Path path, int type) throws RepositoryException {
+ log.debug("pathsDir = "+pathsDir.getAbsolutePath());
+ log.debug("path = "+path);
+ log.debug("path parent = "+path.getParent());
// TODO: Check if leading slash should be removed ...
- File parent = new File(pathsDir + File.separator + path.getParent().toString());
+ File parent = new File(pathsDir + File.separator + path.getParent().toString()); // e.g. access-control/users
if (!parent.exists()) {
log.warn("Directory will be created: " + parent);
parent.mkdirs();
@@ -176,9 +299,35 @@
new File(parent, path.getName()).mkdir();
} else {
try {
- if(!new File(parent, path.getName()).createNewFile()) log.warn("File has not been created: " + new File(parent, path.getName()));
+ if (splitPathEnabled) {
+ // splitted e.g: ab/cd/4.xml
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ log.debug("maybeSplited = "+maybeSplitedPath);
+ String newParent = pathsDir.getAbsolutePath();
+ if (maybeSplitedPath.contains("/")) {
+ // newparent for splitted above would be pathsDir/ab/cd
+ newParent = newParent + maybeSplitedPath.substring(0,maybeSplitedPath.lastIndexOf("/")+1);
+ }
+ String newFileName = new File(maybeSplitedPath).getName();
+ log.debug("new parent = "+newParent);
+ log.debug("new file name = "+newFileName);
+ File newFilePath = new File(newParent , newFileName);
+ new File(newParent).mkdirs();
+ log.debug("new parent exists: "+new File(newParent).exists());
+ boolean created = newFilePath.createNewFile();
+ log.debug("new file exists: "+newFilePath.exists());
+ log.debug("new file is directory: "+newFilePath.isDirectory());
+ if (!created) {
+ log.debug("Maybe file has not been created: " + newFilePath.getAbsolutePath()); // On Mac OSX 10.6, the file gets created even in this case
+ }
+
+ } else {
+ if(!new File(parent, path.getName()).createNewFile()) log.debug("File has not been created: " + new File(parent, path.getName()));
+ }
+
+
} catch (Exception e) {
- log.error(e.getMessage(), e);
+ log.error("Could not create new file!! Exception: "+e.getMessage(), e);
}
}
@@ -186,7 +335,7 @@
}
/**
- *
+ * An exception gets thrown if you call this method because symbolic links are not implemented for virtual file systems!
*/
public void addSymbolicLink(Path path, UID uid) throws RepositoryException {
throw new RepositoryException("Symbolic links not implemented for virtual file system!");
@@ -199,6 +348,9 @@
public ChildrenFilter() {
}
+ /**
+ * @param dir is ignored in this implementation
+ */
public boolean accept(File dir, String name) {
if (VFileSystemMapImpl.this.ignorePath(name)) {
@@ -224,4 +376,20 @@
ignorePatterns = null; // see ignorePath(String)
}
}
+
+ public SplitPathConfig getSplitPathConfig() {
+ return splitPathConfig;
+ }
+
+ public void setSplitPathConfig(SplitPathConfig splitPathConfig) {
+ this.splitPathConfig = splitPathConfig;
+ }
+
+ public boolean isSplitPathEnabled() {
+ return splitPathEnabled;
+ }
+
+ public void setSplitPathEnabled(boolean splitPathEnabled) {
+ this.splitPathEnabled = splitPathEnabled;
+ }
}
Index: src/impl/java/org/wyona/yarep/core/impl/vfs/VFileSystemStorage.java
===================================================================
--- src/impl/java/org/wyona/yarep/core/impl/vfs/VFileSystemStorage.java (revision 57761)
+++ src/impl/java/org/wyona/yarep/core/impl/vfs/VFileSystemStorage.java (working copy)
@@ -1,12 +1,5 @@
package org.wyona.yarep.core.impl.vfs;
-import org.wyona.commons.io.FileUtil;
-import org.wyona.yarep.core.NoSuchNodeException;
-import org.wyona.yarep.core.Path;
-import org.wyona.yarep.core.RepositoryException;
-import org.wyona.yarep.core.Storage;
-import org.wyona.yarep.core.UID;
-
import java.io.File;
import java.io.InputStream;
import java.io.OutputStream;
@@ -15,6 +8,13 @@
import org.apache.avalon.framework.configuration.Configuration;
import org.apache.log4j.Category;
+import org.wyona.commons.io.FileUtil;
+import org.wyona.yarep.core.NoSuchNodeException;
+import org.wyona.yarep.core.Path;
+import org.wyona.yarep.core.RepositoryException;
+import org.wyona.yarep.core.Storage;
+import org.wyona.yarep.core.UID;
+import org.wyona.yarep.impl.repo.vfs.SplitPathConfig;
/**
*
@@ -27,6 +27,12 @@
private String alternative = null;
private String dirListingMimeType = "application/xml";
+ // Configuration parameters of the <splitpath ...> element
+ private SplitPathConfig splitPathConfig = new SplitPathConfig();
+ private boolean splitPathEnabled = false;
+
+
+
/**
* Read VFS Storage configuration
*/
@@ -50,6 +56,28 @@
dirListingMimeType = directoryConfig.getAttribute("mime-type", "application/xml");
log.debug("Mime type of directory listing: " + dirListingMimeType);
}
+
+ // Read the <splitpath> configuration
+ log.debug("Reading Split Path Configuation");
+ Configuration splitConfig = storageConfig.getChild("splitpath", false);
+ if (splitConfig != null) {
+ splitPathEnabled = true;
+ splitPathConfig.setSplitparts(Integer.parseInt(splitConfig.getAttribute("depth", "0")));
+ splitPathConfig.setSplitlength(Integer.parseInt(splitConfig.getAttribute("length", "0")));
+ splitPathConfig.setEscapeChar(splitConfig.getAttribute("escape", "-"));
+
+ int numberOfIncludePaths = splitConfig.getChildren("include").length;
+ int i = 0;
+ if (numberOfIncludePaths > 0) {
+ String[] includepaths = new String[numberOfIncludePaths];
+ for (Configuration include : splitConfig.getChildren("include")) {
+ includepaths[i++] = include.getAttribute("path");
+ }
+ splitPathConfig.setIncludepaths(includepaths);
+ }
+ log.debug("Split Path Configuration DONE");
+ }
+
} catch (Exception e) {
log.error(e);
throw new RepositoryException(e.getMessage(), e);
@@ -60,57 +88,117 @@
*
*/
public Writer getWriter(UID uid, Path path) {
- return new VFileSystemRepositoryWriter(uid, path, contentDir);
+ // TODO We pass null as uid argument because the class anyway does not need it at this moment.
+ // If in future this class is going to process the uid argument too, the splitpathConfig object can be passed to it
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ Path newPath = new Path(maybeSplitedPath);
+ Writer writer = null;
+ try {
+ writer = new VFileSystemRepositoryWriter(null, newPath, contentDir);
+ } catch (Exception e) {
+ }
+ if (writer == null) {
+ writer = new VFileSystemRepositoryWriter(null, path, contentDir);
+ }
+ return writer;
}
/**
- *
+ * @param uid is ignored in this implementation!
*/
public OutputStream getOutputStream(UID uid, Path path) throws RepositoryException {
- return new VFileSystemRepositoryOutputStream(uid, path, contentDir);
+ // TODO We pass null as uid argument because the class anyway does not need it at this moment.
+ // If in future this class is going to process the uid argument too, the splitpathConfig object can be passed to it
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ Path newPath = new Path(maybeSplitedPath);
+ OutputStream out = null;
+ try {
+ out = new VFileSystemRepositoryOutputStream(null, newPath, contentDir);
+ } catch (Exception e) {
+ }
+ if (out == null) {
+ out = new VFileSystemRepositoryOutputStream(null, path, contentDir);
+ }
+ return out;
}
/**
*
*/
public Reader getReader(UID uid, Path path) throws NoSuchNodeException {
- return new VFileSystemRepositoryReader(uid, path, contentDir);
+ // TODO We pass null as uid argument because the class anyway does not need it at this moment.
+ // If in future this class is going to process the uid argument too, the splitpathConfig object can be passed to it
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ Path newPath = new Path(maybeSplitedPath);
+ Reader reader = null;
+ try {
+ reader = new VFileSystemRepositoryReader(null, newPath, contentDir);
+ } catch (Exception e) {
+ }
+ if (reader == null) {
+ reader = new VFileSystemRepositoryReader(null, path, contentDir);
+ }
+ return reader;
}
/**
- *
+ * @param uid is ignored in this implementation!
*/
public InputStream getInputStream(UID uid, Path path) throws RepositoryException {
- return new VFileSystemRepositoryInputStream(uid, path, contentDir, alternative, dirListingMimeType);
+ // TODO: if uid is processed by VFileSystemRepositoryInputStream, the splitPathConfig must be passed to it too.
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig);
+ Path newPath = new Path(maybeSplitedPath);
+ InputStream in = null;
+ try {
+ in = new VFileSystemRepositoryInputStream(null, newPath, contentDir, alternative, dirListingMimeType);
+ } catch (Exception e) {
+ }
+ if (in == null) {
+ in = new VFileSystemRepositoryInputStream(null, path, contentDir, alternative, dirListingMimeType);
+ }
+ return in;
}
/**
- *
+ * @param path is currently ignored!!
*/
public long getLastModified(UID uid, Path path) throws RepositoryException {
- File file = new File(contentDir.getAbsolutePath() + File.separator + uid.toString());
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(uid.toString(), this.splitPathConfig);
+ File file = new File(contentDir.getAbsolutePath() + File.separator + maybeSplitedPath);
+ if (!file.exists()) {
+ file = new File(contentDir.getAbsolutePath() + File.separator + uid.toString());
+ }
return file.lastModified();
}
/**
+ * @param path is currently not used!
* @return Size of file in bytes
*/
public long getSize(UID uid, Path path) throws RepositoryException {
- File file = new File(contentDir.getAbsolutePath() + File.separator + uid.toString());
- return file.length();
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(uid.toString(), this.splitPathConfig);
+ File file = new File(contentDir.getAbsolutePath() + File.separator + maybeSplitedPath);
+ if (!file.exists()) {
+ file = new File(contentDir.getAbsolutePath() + File.separator + uid.toString());
+ }
+ return file.length();
}
/**
- *
+ * @param path is currently not used!
*/
public boolean delete(UID uid, Path path) throws RepositoryException {
- File file = new File(contentDir.getAbsolutePath() + File.separator + uid.toString());
+ String maybeSplitedPath = SplitPathConfig.splitPathIfRequired(uid.toString(), this.splitPathConfig);
+ File file = new File(contentDir.getAbsolutePath() + File.separator + maybeSplitedPath.toString());
+ if (!file.exists()) {
+ file = new File(contentDir.getAbsolutePath() + File.separator + uid.toString());
+ }
log.debug("Try to delete: " + file);
return file.delete();
}
/**
- *
+ * Not implemented at this moment
*/
public String[] getRevisions(UID uid, Path path) throws RepositoryException {
log.warn("Versioning not implemented yet");
@@ -118,16 +206,32 @@
}
/**
- *
+ * Checks the existence via uid first and then via path parameter
*/
public boolean exists(UID uid, Path path) {
+ boolean exists = false;
if (uid != null) {
- return new File(contentDir.getAbsolutePath() + File.separator + uid.toString()).exists();
+ if (splitPathEnabled) {
+ File normalFile = new File(contentDir.getAbsolutePath() + File.separator + uid.toString());
+ File splitFile = new File(contentDir.getAbsolutePath() + File.separator + SplitPathConfig.splitPathIfRequired(uid.toString(), this.splitPathConfig));
+ exists = normalFile.exists() || splitFile.exists();
+ } else {
+ exists = new File(contentDir.getAbsolutePath() + File.separator + uid.toString()).exists();
+ }
+
} else if (path != null) {
log.warn("No UUID specified, hence check path: " + path + " (Content dir: " + contentDir + ")");
- return new File(contentDir.getAbsolutePath() + File.separator + path.toString()).exists();
- } else {
- return false;
- }
+ if (splitPathEnabled) {
+ File normalFile = new File(contentDir.getAbsolutePath() + File.separator + path.toString());
+ File splitFile = new File(contentDir.getAbsolutePath() + File.separator + SplitPathConfig.splitPathIfRequired(path.toString(), this.splitPathConfig));
+ exists = normalFile.exists() || splitFile.exists();
+ } else {
+ exists = new File(contentDir.getAbsolutePath() + File.separator + path.toString()).exists();
+ }
+
+ }
+ return exists;
}
+
+
}
Index: src/impl/java/org/wyona/yarep/core/impl/vfs/VFileSystemRepositoryInputStream.java
===================================================================
--- src/impl/java/org/wyona/yarep/core/impl/vfs/VFileSystemRepositoryInputStream.java (revision 57761)
+++ src/impl/java/org/wyona/yarep/core/impl/vfs/VFileSystemRepositoryInputStream.java (working copy)
@@ -58,7 +58,7 @@
throw new NoSuchNodeException("No such file or directory: " + file);
}
} catch (Exception e) {
- log.error(e);
+ log.warn(e);
in = null;
throw new RepositoryException(e.getMessage(), e);
}
More information about the Yanel-development
mailing list