For a hypergraph $H$ on the vertex set $\{1,\ldots,n\}$, a distribution $D = (D_1,\ldots,D_n)$ over $\{0,1\}^n$ is H-wise independent if every restriction of $D$ to indices which form an edge in $H$ is uniform. This generalizes the notion of $k$-wise independence obtained by taking $H$ to be the complete $n$ vertex $k$-uniform hypergraph. This generalization was studied by Schulman (STOC 1992), who presented constructions of $H$-wise independent distributions that are linear , i.e., the samples are strings of inner products (over $\mathbb{F}_2$) of a fixed set of vectors with a uniformly chosen random vector. Let $\ell(H)$ denote the minimum possible size of a sample space of a uniform $H$-wise independent distribution. The $\ell$ parameter is well understood for the special case of $k$-wise independence. In this work we study the notion of $H$-wise independence and the $\ell$ parameter for general graphs and hypergraphs. For graphs, we show how the $\ell$ parameter relates to standard graph parameters (e.g., clique number, chromatic number, Lovász theta function, minrank). We derive algorithmic and hardness results for this parameter as well as an explicit construction of graphs $G$ for which $\ell(G)$ is exponentially smaller than the size of the sample space of any linear $G$-wise independent distribution. For hypergraphs, we study the problem of testing whether a given distribution is $H$-wise independent, generalizing results of Alon et al. (STOC 2007).